[SPARK-8300] DataFrame hint for broadcast join. by rxin · Pull Request #6751 · apache/spark

rxin · 2015-06-10T22:59:15Z

Users can now do

left.join(broadcast(right), "joinKey")

to give the query planner a hint that "right" DataFrame is small and should be broadcasted.

patmcdonough · 2015-06-10T23:37:17Z

@rxin - This is great. I'm a fan of the alternative syntax we chatted about:

left.join(right.broadcast, "joinKey")

SparkQA · 2015-06-11T00:49:15Z

Test build #34641 has finished for PR 6751 at commit 3ad8de6.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class BroadcastHint(child: LogicalPlan) extends UnaryNode

marmbrus · 2015-06-11T23:19:10Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala

Could we just have this be StatisticsHint and override the statistics? I'm afraid that we are going to forget to add cases in the future as we broadcast more type of joins.

that sounds good.

I'm thinking about this more and actually maybe what we want to do is have this specific node, and a pattern that recognizes canBroadcast. This pattern can check either for a small enough size or this hint and we can use that anywhere we are planning a broadcast operator. The reasoning is that messing with statistics could have other consequences.

+1
Add specific node is also doable, probably the node can be named like Hint in LogicalPlan, and could be more than broadcast, like uniq_key etc.

Any idea?

marmbrus · 2015-06-14T18:27:52Z

Regarding broadcast(...) vs .broadcast, I don't really care too much. However, I think we need to come up with a convention. For example, I was recently confused that its not isNotNull(column) but instead column.isNotNull.

rxin · 2015-06-14T18:36:21Z

We should add isNotNull too.

The isNotNull was from SchemaRDD.

marmbrus · 2015-06-14T19:11:29Z

I'm confused now: both column.isNotNull and column.isNull work today. I was expecting them to be functions instead of being functions that column defines. I'm mostly wondering how we decide what goes where.

rxin · 2015-06-14T19:16:48Z

I meant adding isNotNull as a function, rather than a member method of Column.

Basically most of those Column.function were inherited from SchemaRDD time.

rxin · 2015-06-23T05:47:09Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala

@marmbrus is this what you had in mind?

SparkQA · 2015-06-23T07:48:13Z

Test build #35520 has finished for PR 6751 at commit 953eec2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2015-06-23T08:50:04Z

Merging to master!

sridharsubramanian62 · 2015-10-08T10:19:57Z

How can this hint be used with Spark-SQL ?

Below example does not seem to work.

select a.col1,b.col2 from tableA a left outer join broadcast(tableB) on a.col1 = b.col1

marmbrus · 2015-10-08T16:09:39Z

It can only be used from the dataframe API.
On Oct 8, 2015 3:20 AM, "sridharsubramanian62" notifications@github.com
wrote:

How can this hint be used with Spark-SQL ?

Below example does not seem to work.

select a.col1,b.col2 from tableA a left outer join broadcast(tableB) on
a.col1 = b.col1

—
Reply to this email directly or view it on GitHub
#6751 (comment).

RoiViber · 2015-12-24T13:28:17Z

Using broadcast(...) results in "java.lang.AssertionError: assertion failed: No plan for BroadcastHint" as in https://issues.apache.org/jira/browse/SPARK-12275 so this solution is not working most of the time

fjh100456 · 2017-10-10T11:28:27Z

@rxin @marmbrus
Is there another way to broadcast table with the spark-sql now, except by spark.sql.autoBroadcastJoinThreshold?
And if no, is it a good way to broadcast table by user conf,such as spark.sql.autoBroadcastJoin.mytable=true to broadcast a table named mytable.

rxin · 2017-10-10T20:45:30Z

Isn't the hint available in SQL?

sridharsubramanian62 · 2017-10-10T21:07:20Z

Its available from spark 2.2.0. On Tuesday, October 10, 2017, 1:46:27 PM PDT, Reynold Xin <notifications@github.com> wrote: Isn't the hint available in SQL? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

fjh100456 · 2017-10-11T01:24:50Z

Is there an example?
I use broadcast like the following, but it perform an error.Would you be so kind as to show me an example?

spark-sql> select a.* from tableA a left outer join broadcast(tableB) b on a.a=b.a;
17/10/11 09:06:40 INFO HiveMetaStore: 0: get_table : db=default tbl=tablea
17/10/11 09:06:40 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_table : db=default tbl=tablea

Error in query: cannot resolve 'tableB' given input columns: []; line 1 pos 51;

'Project [ArrayBuffer(a).*]
+- 'Join LeftOuter, ('a.a = 'b.a)
:- SubqueryAlias a
: +- SubqueryAlias tablea
: +- HiveTableRelation default.tablea, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [a#94, b#95, c#96]
+- 'SubqueryAlias b
+- 'UnresolvedTableValuedFunction broadcast, ['tableB]

sridharsubramanian62 · 2017-10-11T03:48:51Z

Let me know if this does not help [SPARK-16475] Broadcast Hint for SQL Queries - ASF JIRA | | | | [SPARK-16475] Broadcast Hint for SQL Queries - ASF JIRA | | | On Tuesday, October 10, 2017, 6:25:57 PM PDT, fjh100456 <notifications@github.com> wrote: Is there an example? I use broadcast like the following, but it perform an error.Would you be so kind as to show me an example? spark-sql> select a.* from tableA a left outer join broadcast(tableB) b on a.a=b.a; 17/10/11 09:06:40 INFO HiveMetaStore: 0: get_table : db=default tbl=tablea 17/10/11 09:06:40 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_table : db=default tbl=tablea Error in query: cannot resolve 'tableB' given input columns: []; line 1 pos 51; 'Project [ArrayBuffer(a).*] +- 'Join LeftOuter, ('a.a = 'b.a) :- SubqueryAlias a : +- SubqueryAlias tablea : +- HiveTableRelation default.tablea, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [a#94, b#95, c#96] +- 'SubqueryAlias b +- 'UnresolvedTableValuedFunction broadcast, ['tableB] — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

fjh100456 · 2017-10-11T06:31:48Z

With /*+ broadcast(table) */, it works well, thank you very much.

marmbrus reviewed Jun 11, 2015
View reviewed changes

rxin added 3 commits June 22, 2015 22:35

[SPARK-8300] DataFrame hint for broadcast join.

8187b88

Fixed import.

88752d8

Code review feedback.

953eec2

rxin force-pushed the broadcastjoin-hint branch from 3ad8de6 to 953eec2 Compare June 23, 2015 05:46

rxin reviewed Jun 23, 2015
View reviewed changes

asfgit closed this in 6ceb169 Jun 23, 2015

rxin deleted the broadcastjoin-hint branch June 23, 2015 18:38

Conversation

rxin commented Jun 10, 2015

Uh oh!

patmcdonough commented Jun 10, 2015

Uh oh!

SparkQA commented Jun 11, 2015

Uh oh!

marmbrus Jun 11, 2015

Choose a reason for hiding this comment

Uh oh!

rxin Jun 11, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus Jun 14, 2015

Choose a reason for hiding this comment

Uh oh!

chenghao-intel Jun 17, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Jun 14, 2015

Uh oh!

rxin commented Jun 14, 2015

Uh oh!

marmbrus commented Jun 14, 2015

Uh oh!

rxin commented Jun 14, 2015

Uh oh!

rxin Jun 23, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus Jun 23, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 23, 2015

Uh oh!

marmbrus commented Jun 23, 2015

Uh oh!

sridharsubramanian62 commented Oct 8, 2015

Uh oh!

marmbrus commented Oct 8, 2015

Uh oh!

RoiViber commented Dec 24, 2015

Uh oh!

fjh100456 commented Oct 10, 2017

Uh oh!

rxin commented Oct 10, 2017

Uh oh!

sridharsubramanian62 commented Oct 10, 2017 via email

Uh oh!

fjh100456 commented Oct 11, 2017

Error in query: cannot resolve 'tableB' given input columns: []; line 1 pos 51;

Uh oh!

sridharsubramanian62 commented Oct 11, 2017 via email

Uh oh!

fjh100456 commented Oct 11, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Error in query: cannot resolve '`tableB`' given input columns: []; line 1 pos 51;